In today's financial securities industry, advances in technology and high-speed, high volume computerized algorithmic trading strategies have combined to make optimized speed critical to success. The markets began as a totally manual process where trade volumes were very small and the time to find a match was measured in minutes. With the advent of electronic alternative trading networks in the 1990's, much more volume was handled by computers without human intervention, which caused the relevant industry measure to move from minutes to seconds, from seconds to milliseconds, and most recently from milliseconds to microseconds.
Not only have the expectations of market processing time changed radically, but the number of total orders that are processed has grown exponentially. This growth in total order volume is driven by high-speed, high volume computerized algorithmic trading models that literally flood the markets by placing thousands of orders and cancels for orders per second as a means to exploit momentary imbalances within and/or between various liquidity destinations/market centers to attempt the execution of “lightning quick” purchases.
Tremendous time and resources are invested by industry participants to minimize the length of time it takes to send transaction information to and from liquidity destinations/market centers. As a result, the outer most limits of performance improvements available from using more powerful computer equipment and telecommunications capabilities are constantly being stretched. In addition, many of the high-speed, high-volume trading strategies include high cancellation rates if all the desired elements of a transaction are not present. This results in as many as one-thousand potential transactions being sent to liquidity destinations/market centers for each transaction that is executed. Moreover, each order placed and cancelled generates additional market data that needs to be analyzed, which in turn may generate more market data, thus creating a feedback loop. As a result, this high volume of messages being sent to and from various liquidity destinations/market centers puts tremendous strain on telecommunications capabilities and creates “queuing” delays as messages are forced to wait until previously transmitted messaging traffic is processed.
Minimizing the time necessary to send transaction information to liquidity destinations/market centers is only part of the challenge. The market factors outlined above also result in a tremendous volume of message traffic emanating from the liquidity destinations/market centers related to the then-current state of the market for each of the numerous stock symbols and associated transactions (“market data”), all with widely varying degrees of importance and relevance.
For instance, PRIOR ART FIG. 1 illustrates how market data is aggregated together indiscriminately and sent down common communications lines in a typical data processing system. This aggregation causes significant delays and latency in delivering the market data necessary for market participants to make trading decisions in a timely manner. In particular, in the typical model illustrated in FIG. 1, the servers receiving the market data cannot keep up with the increasing flow of information. As a result, the incoming data is queued for processing, leading to additional latency. Also, once the queue has reached maximum capacity, market data can be dropped, which causes the loss of data and therefore the true understanding of the status of a symbol.
As shown in FIG. 1, the current industry approach for processing market data is to establish direct connectivity with the different exchanges and bring all the information that the exchanges publish to a group of servers that process the data needed by users or computerized algorithmic trading programs. The key drawbacks with this approach is the extremely high cost of transporting and processing such a huge amount of data, and the ability to do so in a timely fashion. Transmitting this large amount of aggregated, unanalyzed, non-normalized data introduces latency and delays in transmission times. In addition, only after this data is received in a common repository can it then be analyzed, which adds additional time, before it can be sent to end users according to their specific interests.
Further, as illustrated in PRIOR ART FIG. 1, existing market data systems can be analogized to sending a large number of multi-colored glass beads of varying sizes and colors emanating from multiple sources down a common aggregation funnel which is too small to receive all of the beads at one time so the beads queue up. Eventually all the glass beads will pass though the funnel and arrive at a common repository, at which point they can be analyzed and separated by color and size. If a user was interested in only seeing green glass beads of a specific size, they would have to wait for all of the different glass beads to pass through the tunnel and then wait for the proper size green glass beads to be separated from the rest, thereby increasing time for processing the market data.